{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kubeflow E2E MNIST Case: Building, Distributed Training and Serving\n",
    "\n",
    "This example guides you through:\n",
    "  1. Taking an example TensorFlow model and modifying it to support distributed training.\n",
    "  1. Using `Kubeflow Fairing` to build docker image and launch a TFJob to train model.\n",
    "  1. Using `Kubeflow Fairing` to create InferenceService (KFServing) for the trained model.\n",
    "  1. Clean up the TFJob and InferenceService using `kubeflow-tfjob` and `kfserving` SDK client."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements\n",
    "\n",
    "  * The Kubeflow Fairing, TF-Operator and KFServing have been installed in Kubenertes Cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Training Code\n",
    "\n",
    "We modified the [examples](https://github.com/tensorflow/tensorflow/blob/9a24e8acfcd8c9046e1abaac9dbf5e146186f4c2/tensorflow/examples/learn/mnist.py) to be better suited for distributed training and model serving. There is a delta between existing distributed mnist examples and what's needed to run well as a TFJob. The updated training code is [mnist.py](mnist.py). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Required Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: kubeflow-fairing\r\n",
      "Version: 1.0.1\r\n",
      "Summary: Kubeflow Fairing Python SDK.\r\n",
      "Home-page: https://github.com/kubeflow/fairing\r\n",
      "Author: Kubeflow Authors\r\n",
      "Author-email: hejinchi@cn.ibm.com\r\n",
      "License: Apache License Version 2.0\r\n",
      "Location: /opt/python/python36/lib/python3.6/site-packages\r\n",
      "Requires: azure-storage-file, ibm-cos-sdk, google-api-python-client, setuptools, google-cloud-logging, future, oauth2client, docker, notebook, numpy, boto3, azure-mgmt-storage, retrying, google-cloud-storage, kubernetes, kubeflow-pytorchjob, requests, grpcio, python-dateutil, tornado, six, google-auth, kubeflow-tfjob, kfserving, urllib3, httplib2, nbconvert, cloudpickle\r\n",
      "Required-by: virtual-training, remote-training\r\n"
     ]
    }
   ],
   "source": [
    "!pip show kubeflow-fairing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Docker Registry for Kubeflow Fairing\n",
    "\n",
    "* In order to build docker images from your notebook we need a docker registry where the images will be stored\n",
    "\n",
    "**Note:** The below section must be updated to your values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set docker registry to store image.\n",
    "# Ensure you have permission for pushing docker image requests. \n",
    "DOCKER_REGISTRY = 'index.docker.io/jinchi'\n",
    "\n",
    "# Set namespace. Note that the created PVC should be in the namespace.\n",
    "my_namespace = 'hejinchi'\n",
    "# You also can get the default target namepspace using below API.\n",
    "#namespace = fairing_utils.get_default_target_namespace()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create PV/PVC to Store the Exported Model \n",
    "\n",
    "Create Persistent Volume(PV) and Persistent Volume Claim(PVC), the PVC will be used by pods of training and serving for local mode in steps below.\n",
    "\n",
    "**Note:** The below section must be updated to your values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To satify the distributed training, the PVC should be access from all nodes in the cluster.\n",
    "# The example creates a NFS PV to satify that.\n",
    "nfs_server = '172.16.189.69'\n",
    "nfs_path = '/opt/kubeflow/data/mnist'\n",
    "pv_name = 'kubeflow-mnist'\n",
    "pvc_name = 'mnist-pvc'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Optional) Skip below creating PV/PVC step if you set an existing PV and PVC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kubernetes import client as k8s_client\n",
    "from kubernetes import config as k8s_config\n",
    "from kubeflow.fairing.utils import is_running_in_k8s\n",
    "\n",
    "pv_yaml = f'''\n",
    "apiVersion: v1\n",
    "kind: PersistentVolume\n",
    "metadata:\n",
    "  name: {pv_name}\n",
    "spec:\n",
    "  capacity:\n",
    "    storage: 10Gi\n",
    "  accessModes:\n",
    "  - ReadWriteMany\n",
    "  persistentVolumeReclaimPolicy: Retain\n",
    "  nfs:\n",
    "    path: {nfs_path}\n",
    "    server: {nfs_server}\n",
    "'''\n",
    "pvc_yaml = f'''\n",
    "apiVersion: v1\n",
    "kind: PersistentVolumeClaim\n",
    "metadata:\n",
    "  name: {pvc_name}\n",
    "  namespace: {my_namespace}\n",
    "spec:\n",
    "  accessModes:\n",
    "    - ReadWriteMany\n",
    "  storageClassName: \"\"\n",
    "  resources:\n",
    "    requests:\n",
    "      storage: 10Gi\n",
    "'''\n",
    "\n",
    "if is_running_in_k8s():\n",
    "    k8s_config.load_incluster_config()\n",
    "else:\n",
    "    k8s_config.load_kube_config()\n",
    "\n",
    "k8s_core_api = k8s_client.CoreV1Api()\n",
    "k8s_core_api.create_persistent_volume(yaml.safe_load(pv_yaml))\n",
    "k8s_core_api.create_namespaced_persistent_volume_claim(my_namespace, yaml.safe_load(pvc_yaml))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Kubeflow fairing to build the docker image and launch a TFJob for training\n",
    "\n",
    "* Use kubeflow fairing to build a docker image that includes all your dependencies\n",
    "* Launch a TFJob in the on premise cluster to taining model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly set some custom training parameters for TFJob."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_chief = 1 #number of Chief in TFJob \n",
    "num_ps = 1  #number of PS in TFJob \n",
    "num_workers = 2  #number of Worker in TFJob \n",
    "model_dir = \"/mnt\"\n",
    "export_path = \"/mnt/export\" \n",
    "train_steps = \"1000\"\n",
    "batch_size = \"100\"\n",
    "learning_rate = \"0.01\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use Kubeflow Fairing to build a docker image and push to docker registry, and then launch a TFJob in the on-prem cluster for distributed training model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[W 200727 22:57:30 utils:51] The function mounting_pvc has been deprecated,                     please use `volume_mounts`\n",
      "[I 200727 22:57:30 config:134] Using preprocessor: <kubeflow.fairing.preprocessors.base.BasePreProcessor object at 0x7f9cb89424a8>\n",
      "[I 200727 22:57:30 config:136] Using builder: <kubeflow.fairing.builders.docker.docker.DockerBuilder object at 0x7f9cb8942400>\n",
      "[I 200727 22:57:30 config:138] Using deployer: <kubeflow.fairing.deployers.tfjob.tfjob.TfJob object at 0x7f9cb8942470>\n",
      "[I 200727 22:57:30 docker:32] Building image using docker\n",
      "[W 200727 22:57:30 docker:41] Docker command: ['python', '/opt/mnist.py', '--tf-model-dir=/mnt', '--tf-export-dir=/mnt/export', '--tf-train-steps=1000', '--tf-batch-size=100', '--tf-learning-rate=0.01']\n",
      "[I 200727 22:57:30 base:107] Creating docker context: /tmp/fairing_context_yiq6_iyx\n",
      "[W 200727 22:57:30 docker:56] Building docker image index.docker.io/jinchi/mnist:1929A63D...\n",
      "[I 200727 22:57:30 docker:103] Build output: Step 1/5 : FROM tensorflow/tensorflow:1.15.2-py3\n",
      "[I 200727 22:57:30 docker:103] Build output: \n",
      "[I 200727 22:57:30 docker:103] Build output: ---> b2b972268a17\n",
      "[I 200727 22:57:30 docker:103] Build output: Step 2/5 : ADD mnist.py /opt/mnist.py\n",
      "[I 200727 22:57:30 docker:103] Build output: \n",
      "[I 200727 22:57:30 docker:103] Build output: ---> Using cache\n",
      "[I 200727 22:57:30 docker:103] Build output: ---> 57336dff2275\n",
      "[I 200727 22:57:30 docker:103] Build output: Step 3/5 : RUN chmod +x /opt/mnist.py\n",
      "[I 200727 22:57:30 docker:103] Build output: \n",
      "[I 200727 22:57:30 docker:103] Build output: ---> Using cache\n",
      "[I 200727 22:57:30 docker:103] Build output: ---> b4ea1c1ee7a5\n",
      "[I 200727 22:57:30 docker:103] Build output: Step 4/5 : ENTRYPOINT [\"/usr/bin/python\"]\n",
      "[I 200727 22:57:30 docker:103] Build output: \n",
      "[I 200727 22:57:30 docker:103] Build output: ---> Using cache\n",
      "[I 200727 22:57:30 docker:103] Build output: ---> e9382e66f5cf\n",
      "[I 200727 22:57:30 docker:103] Build output: Step 5/5 : CMD [\"/opt/mnist.py\"]\n",
      "[I 200727 22:57:30 docker:103] Build output: \n",
      "[I 200727 22:57:30 docker:103] Build output: ---> Using cache\n",
      "[I 200727 22:57:30 docker:103] Build output: ---> bf050a423336\n",
      "[I 200727 22:57:30 docker:103] Push finished: {'ID': 'sha256:bf050a423336ba4593ff297ed2c1f209c24d1f1c709465028a6dd4863a50b169'}\n",
      "[I 200727 22:57:30 docker:103] Build output: Successfully built bf050a423336\n",
      "[I 200727 22:57:30 docker:103] Build output: Successfully tagged jinchi/mnist:1929A63D\n",
      "[W 200727 22:57:30 docker:70] Publishing image index.docker.io/jinchi/mnist:1929A63D...\n",
      "[I 200727 22:57:30 docker:103] Push output: The push refers to repository [docker.io/jinchi/mnist] None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Preparing None\n",
      "[I 200727 22:57:31 docker:103] Push output: Waiting None\n",
      "[I 200727 22:57:31 docker:103] Push output: Waiting None\n",
      "[I 200727 22:57:31 docker:103] Push output: Waiting None\n",
      "[I 200727 22:57:31 docker:103] Push output: Waiting None\n",
      "[I 200727 22:57:31 docker:103] Push output: Waiting None\n",
      "[I 200727 22:57:31 docker:103] Push output: Waiting None\n",
      "[I 200727 22:57:31 docker:103] Push output: Waiting None\n",
      "[I 200727 22:57:31 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:31 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:31 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:31 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:31 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:32 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:32 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:32 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:32 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:32 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:32 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:32 docker:103] Push output: Layer already exists None\n",
      "[I 200727 22:57:33 docker:103] Push output: 1929A63D: digest: sha256:16f97471035ab00125bbc9197c893180c455a8d8f05a67553de50834c7b100d3 size: 2828 None\n",
      "[I 200727 22:57:33 docker:103] Push finished: {'Tag': '1929A63D', 'Digest': 'sha256:16f97471035ab00125bbc9197c893180c455a8d8f05a67553de50834c7b100d3', 'Size': 2828}\n",
      "[W 200727 22:57:33 job:101] The tfjob mnist-training-820c launched.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(<kubeflow.fairing.preprocessors.base.BasePreProcessor at 0x7f9cb89424a8>,\n",
       " <kubeflow.fairing.builders.docker.docker.DockerBuilder at 0x7f9cb8942400>,\n",
       " <kubeflow.fairing.deployers.tfjob.tfjob.TfJob at 0x7f9cb8942470>)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import uuid\n",
    "from kubeflow import fairing   \n",
    "from kubeflow.fairing.kubernetes.utils import mounting_pvc\n",
    "\n",
    "tfjob_name = f'mnist-training-{uuid.uuid4().hex[:4]}'\n",
    "\n",
    "output_map =  {\n",
    "    \"Dockerfile\": \"Dockerfile\",\n",
    "    \"mnist.py\": \"mnist.py\"\n",
    "}\n",
    "\n",
    "command=[\"python\",\n",
    "         \"/opt/mnist.py\",\n",
    "         \"--tf-model-dir=\" + model_dir,\n",
    "         \"--tf-export-dir=\" + export_path,\n",
    "         \"--tf-train-steps=\" + train_steps,\n",
    "         \"--tf-batch-size=\" + batch_size,\n",
    "         \"--tf-learning-rate=\" + learning_rate]\n",
    "\n",
    "fairing.config.set_preprocessor('python', command=command, path_prefix=\"/app\", output_map=output_map)\n",
    "fairing.config.set_builder(name='docker', registry=DOCKER_REGISTRY, base_image=\"\",\n",
    "                           image_name=\"mnist\", dockerfile_path=\"Dockerfile\")\n",
    "fairing.config.set_deployer(name='tfjob', namespace=my_namespace, stream_log=False, job_name=tfjob_name,\n",
    "                            chief_count=num_chief, worker_count=num_workers, ps_count=num_ps, \n",
    "                            pod_spec_mutators=[mounting_pvc(pvc_name=pvc_name, pvc_mount_path=model_dir)])\n",
    "fairing.config.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the Created TFJobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'apiVersion': 'kubeflow.org/v1',\n",
       " 'kind': 'TFJob',\n",
       " 'metadata': {'creationTimestamp': '2020-07-28T05:57:33Z',\n",
       "  'generateName': 'fairing-tfjob-',\n",
       "  'generation': 1,\n",
       "  'labels': {'fairing-deployer': 'tfjob',\n",
       "   'fairing-id': '3b20a9c4-d097-11ea-a7a8-00163e01bd45'},\n",
       "  'name': 'mnist-training-820c',\n",
       "  'namespace': 'hejinchi',\n",
       "  'resourceVersion': '102422171',\n",
       "  'selfLink': '/apis/kubeflow.org/v1/namespaces/hejinchi/tfjobs/mnist-training-820c',\n",
       "  'uid': '637adb22-ae73-449d-a3be-7df1410d5ac7'},\n",
       " 'spec': {'tfReplicaSpecs': {'Chief': {'replicas': 1,\n",
       "    'template': {'metadata': {'annotations': {'sidecar.istio.io/inject': 'false'},\n",
       "      'labels': {'fairing-deployer': 'tfjob',\n",
       "       'fairing-id': '3b20a9c4-d097-11ea-a7a8-00163e01bd45'},\n",
       "      'name': 'fairing-deployer'},\n",
       "     'spec': {'containers': [{'command': ['python',\n",
       "         '/opt/mnist.py',\n",
       "         '--tf-model-dir=/mnt',\n",
       "         '--tf-export-dir=/mnt/export',\n",
       "         '--tf-train-steps=1000',\n",
       "         '--tf-batch-size=100',\n",
       "         '--tf-learning-rate=0.01'],\n",
       "        'env': [{'name': 'FAIRING_RUNTIME', 'value': '1'}],\n",
       "        'image': 'index.docker.io/jinchi/mnist:1929A63D',\n",
       "        'name': 'tensorflow',\n",
       "        'securityContext': {'runAsUser': 0},\n",
       "        'volumeMounts': [{'mountPath': '/mnt',\n",
       "          'name': 'fairing-volume-mnist-pvc'}],\n",
       "        'workingDir': '/app'}],\n",
       "      'restartPolicy': 'Never',\n",
       "      'volumes': [{'name': 'fairing-volume-mnist-pvc',\n",
       "        'persistentVolumeClaim': {'claimName': 'mnist-pvc'}}]}}},\n",
       "   'PS': {'replicas': 1,\n",
       "    'template': {'metadata': {'annotations': {'sidecar.istio.io/inject': 'false'},\n",
       "      'labels': {'fairing-deployer': 'tfjob',\n",
       "       'fairing-id': '3b20a9c4-d097-11ea-a7a8-00163e01bd45'},\n",
       "      'name': 'fairing-deployer'},\n",
       "     'spec': {'containers': [{'command': ['python',\n",
       "         '/opt/mnist.py',\n",
       "         '--tf-model-dir=/mnt',\n",
       "         '--tf-export-dir=/mnt/export',\n",
       "         '--tf-train-steps=1000',\n",
       "         '--tf-batch-size=100',\n",
       "         '--tf-learning-rate=0.01'],\n",
       "        'env': [{'name': 'FAIRING_RUNTIME', 'value': '1'}],\n",
       "        'image': 'index.docker.io/jinchi/mnist:1929A63D',\n",
       "        'name': 'tensorflow',\n",
       "        'securityContext': {'runAsUser': 0},\n",
       "        'volumeMounts': [{'mountPath': '/mnt',\n",
       "          'name': 'fairing-volume-mnist-pvc'}],\n",
       "        'workingDir': '/app'}],\n",
       "      'restartPolicy': 'Never',\n",
       "      'volumes': [{'name': 'fairing-volume-mnist-pvc',\n",
       "        'persistentVolumeClaim': {'claimName': 'mnist-pvc'}}]}}},\n",
       "   'Worker': {'replicas': 2,\n",
       "    'template': {'metadata': {'annotations': {'sidecar.istio.io/inject': 'false'},\n",
       "      'labels': {'fairing-deployer': 'tfjob',\n",
       "       'fairing-id': '3b20a9c4-d097-11ea-a7a8-00163e01bd45'},\n",
       "      'name': 'fairing-deployer'},\n",
       "     'spec': {'containers': [{'command': ['python',\n",
       "         '/opt/mnist.py',\n",
       "         '--tf-model-dir=/mnt',\n",
       "         '--tf-export-dir=/mnt/export',\n",
       "         '--tf-train-steps=1000',\n",
       "         '--tf-batch-size=100',\n",
       "         '--tf-learning-rate=0.01'],\n",
       "        'env': [{'name': 'FAIRING_RUNTIME', 'value': '1'}],\n",
       "        'image': 'index.docker.io/jinchi/mnist:1929A63D',\n",
       "        'name': 'tensorflow',\n",
       "        'securityContext': {'runAsUser': 0},\n",
       "        'volumeMounts': [{'mountPath': '/mnt',\n",
       "          'name': 'fairing-volume-mnist-pvc'}],\n",
       "        'workingDir': '/app'}],\n",
       "      'restartPolicy': 'Never',\n",
       "      'volumes': [{'name': 'fairing-volume-mnist-pvc',\n",
       "        'persistentVolumeClaim': {'claimName': 'mnist-pvc'}}]}}}}},\n",
       " 'status': {'conditions': [{'lastTransitionTime': '2020-07-28T05:57:33Z',\n",
       "    'lastUpdateTime': '2020-07-28T05:57:33Z',\n",
       "    'message': 'TFJob mnist-training-820c is created.',\n",
       "    'reason': 'TFJobCreated',\n",
       "    'status': 'True',\n",
       "    'type': 'Created'},\n",
       "   {'lastTransitionTime': '2020-07-28T05:57:37Z',\n",
       "    'lastUpdateTime': '2020-07-28T05:57:37Z',\n",
       "    'message': 'TFJob mnist-training-820c is running.',\n",
       "    'reason': 'TFJobRunning',\n",
       "    'status': 'True',\n",
       "    'type': 'Running'}],\n",
       "  'replicaStatuses': {'Chief': {'active': 1},\n",
       "   'PS': {'active': 1},\n",
       "   'Worker': {'active': 2}},\n",
       "  'startTime': '2020-07-28T05:57:34Z'}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from kubeflow.tfjob import TFJobClient\n",
    "tfjob_client = TFJobClient()\n",
    "\n",
    "tfjob_client.get(tfjob_name, namespace=my_namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait For the Training Job to finish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                           STATE                TIME                          \n",
      "mnist-training-820c            Running              2020-07-28T05:57:37Z          \n",
      "mnist-training-820c            Succeeded            2020-07-28T05:57:43Z          \n"
     ]
    }
   ],
   "source": [
    "tfjob_client.wait_for_job(tfjob_name, namespace=my_namespace, watch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check if the TFJob succeeded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfjob_client.is_job_succeeded(tfjob_name, namespace=my_namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the Training Logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 200727 22:57:51 tf_job_client:386] The logs of Pod mnist-training-820c-chief-0:\n",
      "     WARNING:tensorflow:From /opt/mnist.py:237: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.\n",
      "    \n",
      "    WARNING:tensorflow:From /opt/mnist.py:155: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
      "    \n",
      "    W0728 05:57:37.944859 140369861093184 module_wrapper.py:139] From /opt/mnist.py:155: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
      "    \n",
      "    WARNING:tensorflow:From /opt/mnist.py:155: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
      "    \n",
      "    W0728 05:57:37.945076 140369861093184 module_wrapper.py:139] From /opt/mnist.py:155: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
      "    \n",
      "    WARNING:tensorflow:From /opt/mnist.py:160: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
      "    \n",
      "    W0728 05:57:37.946639 140369861093184 module_wrapper.py:139] From /opt/mnist.py:160: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
      "    \n",
      "    INFO:tensorflow:TF_CONFIG {\"cluster\":{\"chief\":[\"mnist-training-820c-chief-0.hejinchi.svc:2222\"],\"ps\":[\"mnist-training-820c-ps-0.hejinchi.svc:2222\"],\"worker\":[\"mnist-training-820c-worker-0.hejinchi.svc:2222\",\"mnist-training-820c-worker-1.hejinchi.svc:2222\"]},\"task\":{\"type\":\"chief\",\"index\":0},\"environment\":\"cloud\"}\n",
      "    I0728 05:57:37.946778 140369861093184 mnist.py:160] TF_CONFIG {\"cluster\":{\"chief\":[\"mnist-training-820c-chief-0.hejinchi.svc:2222\"],\"ps\":[\"mnist-training-820c-ps-0.hejinchi.svc:2222\"],\"worker\":[\"mnist-training-820c-worker-0.hejinchi.svc:2222\",\"mnist-training-820c-worker-1.hejinchi.svc:2222\"]},\"task\":{\"type\":\"chief\",\"index\":0},\"environment\":\"cloud\"}\n",
      "    INFO:tensorflow:cluster={'chief': ['mnist-training-820c-chief-0.hejinchi.svc:2222'], 'ps': ['mnist-training-820c-ps-0.hejinchi.svc:2222'], 'worker': ['mnist-training-820c-worker-0.hejinchi.svc:2222', 'mnist-training-820c-worker-1.hejinchi.svc:2222']} job_name=chief task_index=0\n",
      "    I0728 05:57:37.947352 140369861093184 mnist.py:166] cluster={'chief': ['mnist-training-820c-chief-0.hejinchi.svc:2222'], 'ps': ['mnist-training-820c-ps-0.hejinchi.svc:2222'], 'worker': ['mnist-training-820c-worker-0.hejinchi.svc:2222', 'mnist-training-820c-worker-1.hejinchi.svc:2222']} job_name=chief task_index=0\n",
      "    INFO:tensorflow:Will export model\n",
      "    I0728 05:57:37.947452 140369861093184 mnist.py:171] Will export model\n",
      "    WARNING:tensorflow:\n",
      "    The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "    For more information, please see:\n",
      "      * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "      * https://github.com/tensorflow/addons\n",
      "      * https://github.com/tensorflow/io (for I/O related ops)\n",
      "    If you depend on functionality not listed there, please file an issue.\n",
      "    \n",
      "    W0728 05:57:37.947560 140369861093184 lazy_loader.py:50] \n",
      "    The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "    For more information, please see:\n",
      "      * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "      * https://github.com/tensorflow/addons\n",
      "      * https://github.com/tensorflow/io (for I/O related ops)\n",
      "    If you depend on functionality not listed there, please file an issue.\n",
      "    \n",
      "    WARNING:tensorflow:From /opt/mnist.py:176: load_mnist (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "    W0728 05:57:38.392488 140369861093184 deprecation.py:323] From /opt/mnist.py:176: load_mnist (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:300: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "    W0728 05:57:38.392761 140369861093184 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:300: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please write your own downloading logic.\n",
      "    W0728 05:57:38.392931 140369861093184 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please write your own downloading logic.\n",
      "    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use urllib or similar directly.\n",
      "    W0728 05:57:38.393539 140369861093184 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use urllib or similar directly.\n",
      "    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use tf.data to implement this functionality.\n",
      "    W0728 05:57:39.483829 140369861093184 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use tf.data to implement this functionality.\n",
      "    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use tf.data to implement this functionality.\n",
      "    W0728 05:57:40.142219 140369861093184 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use tf.data to implement this functionality.\n",
      "    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "    W0728 05:57:41.429694 140369861093184 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "    WARNING:tensorflow:From /opt/mnist.py:177: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.\n",
      "    \n",
      "    W0728 05:57:41.635517 140369861093184 module_wrapper.py:139] From /opt/mnist.py:177: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.\n",
      "    \n",
      "    WARNING:tensorflow:From /opt/mnist.py:177: The name tf.estimator.inputs.numpy_input_fn is deprecated. Please use tf.compat.v1.estimator.inputs.numpy_input_fn instead.\n",
      "    \n",
      "    W0728 05:57:41.635798 140369861093184 module_wrapper.py:139] From /opt/mnist.py:177: The name tf.estimator.inputs.numpy_input_fn is deprecated. Please use tf.compat.v1.estimator.inputs.numpy_input_fn instead.\n",
      "    \n",
      "    INFO:tensorflow:TF_CONFIG environment variable: {'cluster': {'chief': ['mnist-training-820c-chief-0.hejinchi.svc:2222'], 'ps': ['mnist-training-820c-ps-0.hejinchi.svc:2222'], 'worker': ['mnist-training-820c-worker-0.hejinchi.svc:2222', 'mnist-training-820c-worker-1.hejinchi.svc:2222']}, 'task': {'type': 'chief', 'index': 0}, 'environment': 'cloud'}\n",
      "    I0728 05:57:41.636092 140369861093184 run_config.py:535] TF_CONFIG environment variable: {'cluster': {'chief': ['mnist-training-820c-chief-0.hejinchi.svc:2222'], 'ps': ['mnist-training-820c-ps-0.hejinchi.svc:2222'], 'worker': ['mnist-training-820c-worker-0.hejinchi.svc:2222', 'mnist-training-820c-worker-1.hejinchi.svc:2222']}, 'task': {'type': 'chief', 'index': 0}, 'environment': 'cloud'}\n",
      "    INFO:tensorflow:Using config: {'_model_dir': '/mnt', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': device_filters: \"/job:ps\"\n",
      "    device_filters: \"/job:chief\"\n",
      "    allow_soft_placement: true\n",
      "    graph_options {\n",
      "      rewrite_options {\n",
      "        meta_optimizer_iterations: ONE\n",
      "      }\n",
      "    }\n",
      "    , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7faa35f47048>, '_task_type': 'chief', '_task_id': 0, '_evaluation_master': '', '_master': 'grpc://mnist-training-820c-chief-0.hejinchi.svc:2222', '_num_ps_replicas': 1, '_num_worker_replicas': 3, '_global_id_in_cluster': 0, '_is_chief': True}\n",
      "    I0728 05:57:41.637499 140369861093184 estimator.py:212] Using config: {'_model_dir': '/mnt', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': device_filters: \"/job:ps\"\n",
      "    device_filters: \"/job:chief\"\n",
      "    allow_soft_placement: true\n",
      "    graph_options {\n",
      "      rewrite_options {\n",
      "        meta_optimizer_iterations: ONE\n",
      "      }\n",
      "    }\n",
      "    , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7faa35f47048>, '_task_type': 'chief', '_task_id': 0, '_evaluation_master': '', '_master': 'grpc://mnist-training-820c-chief-0.hejinchi.svc:2222', '_num_ps_replicas': 1, '_num_worker_replicas': 3, '_global_id_in_cluster': 0, '_is_chief': True}\n",
      "    INFO:tensorflow:Not using Distribute Coordinator.\n",
      "    I0728 05:57:41.638131 140369861093184 estimator_training.py:186] Not using Distribute Coordinator.\n",
      "    INFO:tensorflow:Start Tensorflow server.\n",
      "    I0728 05:57:41.638589 140369861093184 training.py:744] Start Tensorflow server.\n",
      "    2020-07-28 05:57:41.639064: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA\n",
      "    2020-07-28 05:57:41.646975: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400000000 Hz\n",
      "    2020-07-28 05:57:41.648339: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3fcbd50 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "    2020-07-28 05:57:41.648378: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "    2020-07-28 05:57:41.651408: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:258] Initialize GrpcChannelCache for job chief -> {0 -> localhost:2222}\n",
      "    2020-07-28 05:57:41.651432: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:258] Initialize GrpcChannelCache for job ps -> {0 -> mnist-training-820c-ps-0.hejinchi.svc:2222}\n",
      "    2020-07-28 05:57:41.651440: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:258] Initialize GrpcChannelCache for job worker -> {0 -> mnist-training-820c-worker-0.hejinchi.svc:2222, 1 -> mnist-training-820c-worker-1.hejinchi.svc:2222}\n",
      "    2020-07-28 05:57:41.655397: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:365] Started server with target: grpc://localhost:2222\n",
      "    INFO:tensorflow:Skipping training since max_steps has already saved.\n",
      "    I0728 05:57:41.681308 140369861093184 estimator.py:363] Skipping training since max_steps has already saved.\n",
      "    WARNING:tensorflow:From /opt/mnist.py:232: Estimator.export_savedmodel (from tensorflow_estimator.python.estimator.estimator) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    This function has been renamed, use `export_saved_model` instead.\n",
      "    W0728 05:57:41.681692 140369861093184 deprecation.py:323] From /opt/mnist.py:232: Estimator.export_savedmodel (from tensorflow_estimator.python.estimator.estimator) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    This function has been renamed, use `export_saved_model` instead.\n",
      "    WARNING:tensorflow:From /opt/mnist.py:145: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "    \n",
      "    W0728 05:57:41.717176 140369861093184 module_wrapper.py:139] From /opt/mnist.py:145: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "    \n",
      "    INFO:tensorflow:Calling model_fn.\n",
      "    I0728 05:57:41.720393 140369861093184 estimator.py:1148] Calling model_fn.\n",
      "    WARNING:tensorflow:From /opt/mnist.py:76: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
      "    \n",
      "    W0728 05:57:41.727399 140369861093184 module_wrapper.py:139] From /opt/mnist.py:76: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
      "    \n",
      "    WARNING:tensorflow:From /opt/mnist.py:82: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Use `tf.keras.layers.Conv2D` instead.\n",
      "    W0728 05:57:41.728867 140369861093184 deprecation.py:323] From /opt/mnist.py:82: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Use `tf.keras.layers.Conv2D` instead.\n",
      "    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use `layer.__call__` method instead.\n",
      "    W0728 05:57:41.730894 140369861093184 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Please use `layer.__call__` method instead.\n",
      "    WARNING:tensorflow:From /opt/mnist.py:84: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Use keras.layers.MaxPooling2D instead.\n",
      "    W0728 05:57:41.771493 140369861093184 deprecation.py:323] From /opt/mnist.py:84: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Use keras.layers.MaxPooling2D instead.\n",
      "    WARNING:tensorflow:From /opt/mnist.py:100: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Use keras.layers.Dense instead.\n",
      "    W0728 05:57:41.797417 140369861093184 deprecation.py:323] From /opt/mnist.py:100: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Use keras.layers.Dense instead.\n",
      "    WARNING:tensorflow:From /opt/mnist.py:104: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Use keras.layers.dropout instead.\n",
      "    W0728 05:57:41.815707 140369861093184 deprecation.py:323] From /opt/mnist.py:104: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    Use keras.layers.dropout instead.\n",
      "    INFO:tensorflow:Done calling model_fn.\n",
      "    I0728 05:57:41.837737 140369861093184 estimator.py:1150] Done calling model_fn.\n",
      "    WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "    W0728 05:57:41.838060 140369861093184 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "    Instructions for updating:\n",
      "    This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "    INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "    I0728 05:57:41.838558 140369861093184 export_utils.py:170] Signatures INCLUDED in export for Classify: None\n",
      "    INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "    I0728 05:57:41.838679 140369861093184 export_utils.py:170] Signatures INCLUDED in export for Regress: None\n",
      "    INFO:tensorflow:Signatures INCLUDED in export for Predict: ['classes', 'serving_default']\n",
      "    I0728 05:57:41.838748 140369861093184 export_utils.py:170] Signatures INCLUDED in export for Predict: ['classes', 'serving_default']\n",
      "    INFO:tensorflow:Signatures INCLUDED in export for Train: None\n",
      "    I0728 05:57:41.838858 140369861093184 export_utils.py:170] Signatures INCLUDED in export for Train: None\n",
      "    INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n",
      "    I0728 05:57:41.838952 140369861093184 export_utils.py:170] Signatures INCLUDED in export for Eval: None\n",
      "    INFO:tensorflow:Restoring parameters from /mnt/model.ckpt-2004\n",
      "    I0728 05:57:41.871512 140369861093184 saver.py:1284] Restoring parameters from /mnt/model.ckpt-2004\n",
      "    INFO:tensorflow:Assets added to graph.\n",
      "    I0728 05:57:41.919574 140369861093184 builder_impl.py:665] Assets added to graph.\n",
      "    INFO:tensorflow:No assets to write.\n",
      "    I0728 05:57:41.919970 140369861093184 builder_impl.py:460] No assets to write.\n",
      "    INFO:tensorflow:SavedModel written to: /mnt/export/temp-b'1595915861'/saved_model.pb\n",
      "    I0728 05:57:42.070577 140369861093184 builder_impl.py:425] SavedModel written to: /mnt/export/temp-b'1595915861'/saved_model.pb\n",
      "    Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "    Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "    Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "    Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "    Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "    Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "    Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "    Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n",
      "    Train and evaluate\n",
      "    Training done\n",
      "    Export saved model\n",
      "    Done exporting the model\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "tfjob_client.get_logs(tfjob_name, namespace=my_namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy Service using KFServing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                 READY      DEFAULT_TRAFFIC CANARY_TRAFFIC  URL                                               \n",
      "mnist-service-f129   Unknown                                                                                      \n",
      "mnist-service-f129   False                                                                                        \n",
      "mnist-service-f129   False                                                                                        \n",
      "mnist-service-f129   False                                                                                        \n",
      "mnist-service-f129   False                                                                                        \n",
      "mnist-service-f129   True       100                             http://mnist-service-f129.hejinchi.example.com/...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 200727 22:58:15 kfserving:127] Deployed the InferenceService mnist-service-f129 successfully.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'mnist-service-f129'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from kubeflow.fairing.deployers.kfserving.kfserving import KFServing\n",
    "isvc_name = f'mnist-service-{uuid.uuid4().hex[:4]}'\n",
    "isvc = KFServing('tensorflow', namespace=my_namespace, isvc_name=isvc_name,\n",
    "                 default_storage_uri='pvc://' + pvc_name + '/export')\n",
    "isvc.deploy(isvc.generate_isvc())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the InferenceService"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'apiVersion': 'serving.kubeflow.org/v1alpha2',\n",
       " 'items': [{'apiVersion': 'serving.kubeflow.org/v1alpha2',\n",
       "   'kind': 'InferenceService',\n",
       "   'metadata': {'creationTimestamp': '2020-07-28T02:39:33Z',\n",
       "    'generateName': 'fairing-kfserving-',\n",
       "    'generation': 5,\n",
       "    'name': 'mnist-service-4db9',\n",
       "    'namespace': 'hejinchi',\n",
       "    'resourceVersion': '102344640',\n",
       "    'selfLink': '/apis/serving.kubeflow.org/v1alpha2/namespaces/hejinchi/inferenceservices/mnist-service-4db9',\n",
       "    'uid': '37f5c4c6-86ed-4a71-9572-810e03a1413f'},\n",
       "   'spec': {'default': {'predictor': {'tensorflow': {'resources': {'limits': {'cpu': '1',\n",
       "         'memory': '2Gi'},\n",
       "        'requests': {'cpu': '1', 'memory': '2Gi'}},\n",
       "       'runtimeVersion': '1.14.0',\n",
       "       'storageUri': 'pvc://mnist-pvc/export'}}}},\n",
       "   'status': {'canary': {},\n",
       "    'conditions': [{'lastTransitionTime': '2020-07-28T02:40:03Z',\n",
       "      'status': 'True',\n",
       "      'type': 'DefaultPredictorReady'},\n",
       "     {'lastTransitionTime': '2020-07-28T02:40:03Z',\n",
       "      'status': 'True',\n",
       "      'type': 'Ready'},\n",
       "     {'lastTransitionTime': '2020-07-28T02:40:03Z',\n",
       "      'status': 'True',\n",
       "      'type': 'RoutesReady'}],\n",
       "    'default': {'predictor': {'host': 'mnist-service-4db9-predictor-default.hejinchi.example.com',\n",
       "      'name': 'mnist-service-4db9-predictor-default-lrfs8'}},\n",
       "    'traffic': 100,\n",
       "    'url': 'http://mnist-service-4db9.hejinchi.example.com/v1/models/mnist-service-4db9'}},\n",
       "  {'apiVersion': 'serving.kubeflow.org/v1alpha2',\n",
       "   'kind': 'InferenceService',\n",
       "   'metadata': {'creationTimestamp': '2020-07-28T05:57:59Z',\n",
       "    'generateName': 'fairing-kfserving-',\n",
       "    'generation': 6,\n",
       "    'name': 'mnist-service-f129',\n",
       "    'namespace': 'hejinchi',\n",
       "    'resourceVersion': '102422525',\n",
       "    'selfLink': '/apis/serving.kubeflow.org/v1alpha2/namespaces/hejinchi/inferenceservices/mnist-service-f129',\n",
       "    'uid': 'bde8a9ca-e7dc-4430-abd1-7eae0c298311'},\n",
       "   'spec': {'default': {'predictor': {'tensorflow': {'resources': {'limits': {'cpu': '1',\n",
       "         'memory': '2Gi'},\n",
       "        'requests': {'cpu': '1', 'memory': '2Gi'}},\n",
       "       'runtimeVersion': '1.14.0',\n",
       "       'storageUri': 'pvc://mnist-pvc/export'}}}},\n",
       "   'status': {'canary': {},\n",
       "    'conditions': [{'lastTransitionTime': '2020-07-28T05:58:15Z',\n",
       "      'status': 'True',\n",
       "      'type': 'DefaultPredictorReady'},\n",
       "     {'lastTransitionTime': '2020-07-28T05:58:15Z',\n",
       "      'status': 'True',\n",
       "      'type': 'Ready'},\n",
       "     {'lastTransitionTime': '2020-07-28T05:58:15Z',\n",
       "      'status': 'True',\n",
       "      'type': 'RoutesReady'}],\n",
       "    'default': {'predictor': {'host': 'mnist-service-f129-predictor-default.hejinchi.example.com',\n",
       "      'name': 'mnist-service-f129-predictor-default-mqsch'}},\n",
       "    'traffic': 100,\n",
       "    'url': 'http://mnist-service-f129.hejinchi.example.com/v1/models/mnist-service-f129'}}],\n",
       " 'kind': 'InferenceServiceList',\n",
       " 'metadata': {'continue': '',\n",
       "  'resourceVersion': '102422548',\n",
       "  'selfLink': '/apis/serving.kubeflow.org/v1alpha2/namespaces/hejinchi/inferenceservices'}}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from kfserving import KFServingClient\n",
    "kfserving_client = KFServingClient()\n",
    "kfserving_client.get(namespace=my_namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the InferenceService and Service Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MNIST Service Endpoint: http://mnist-service-f129.hejinchi.example.com/v1/models/mnist-service-f129\n"
     ]
    }
   ],
   "source": [
    "mnist_isvc = kfserving_client.get(isvc_name, namespace=my_namespace)\n",
    "mnist_isvc_name = mnist_isvc['metadata']['name']\n",
    "mnist_isvc_endpoint = mnist_isvc['status'].get('url', '')\n",
    "print(\"MNIST Service Endpoint: \" + mnist_isvc_endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run a prediction to the InferenceService"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* About to connect() to 10.110.51.90 port 80 (#0)\n",
      "*   Trying 10.110.51.90...\n",
      "* Connected to 10.110.51.90 (10.110.51.90) port 80 (#0)\n",
      "> POST /v1/models/mnist-service-f129:predict HTTP/1.1\n",
      "> User-Agent: curl/7.29.0\n",
      "> Accept: */*\n",
      "> Host: mnist-service-f129.hejinchi.example.com\n",
      "> Content-Length: 2052\n",
      "> Content-Type: application/x-www-form-urlencoded\n",
      "> Expect: 100-continue\n",
      "> \n",
      "< HTTP/1.1 100 Continue\n",
      "< HTTP/1.1 200 OK\n",
      "< content-length: 257\n",
      "< content-type: application/json\n",
      "< date: Tue, 28 Jul 2020 05:58:38 GMT\n",
      "< x-envoy-upstream-service-time: 314\n",
      "< server: istio-envoy\n",
      "< \n",
      "{\n",
      "    \"predictions\": [\n",
      "        {\n",
      "            \"classes\": 8,\n",
      "            \"predictions\": [2.49753812e-05, 9.58313558e-06, 0.000792403473, 0.0170188081, 3.87205364e-05, 0.00188907969, 1.35709442e-05, 2.67499416e-07, 0.980212331, 2.4302895e-07]\n",
      "        }\n",
      "    ]\n",
      "* Connection #0 to host 10.110.51.90 left intact\n",
      "}"
     ]
    }
   ],
   "source": [
    "ISTIO_CLUSTER_IP=!kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}'\n",
    "CLUSTER_IP=ISTIO_CLUSTER_IP[0]\n",
    "MODEL_HOST=f\"Host: {mnist_isvc_name}.{my_namespace}.example.com\"\n",
    "!curl -v -H \"{MODEL_HOST}\" http://{CLUSTER_IP}/v1/models/{mnist_isvc_name}:predict -d @./input.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean Up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete the TFJob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'kind': 'Status',\n",
       " 'apiVersion': 'v1',\n",
       " 'metadata': {},\n",
       " 'status': 'Success',\n",
       " 'details': {'name': 'mnist-training-820c',\n",
       "  'group': 'kubeflow.org',\n",
       "  'kind': 'tfjobs',\n",
       "  'uid': '637adb22-ae73-449d-a3be-7df1410d5ac7'}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfjob_client.delete(tfjob_name, namespace=my_namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete the InferenceService."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'kind': 'Status',\n",
       " 'apiVersion': 'v1',\n",
       " 'metadata': {},\n",
       " 'status': 'Success',\n",
       " 'details': {'name': 'mnist-service-f129',\n",
       "  'group': 'serving.kubeflow.org',\n",
       "  'kind': 'inferenceservices',\n",
       "  'uid': 'bde8a9ca-e7dc-4430-abd1-7eae0c298311'}}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kfserving_client.delete(isvc_name, namespace=my_namespace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
